Papers with evaluation measures
SUMMARY WORKBENCH: Unifying Application and Evaluation of Text Summarization Models (2022.emnlp-demos)
Copied to clipboard
| Challenge: | Summary Workbench is a tool for developing and evaluating text summarization models. |
| Approach: | They propose a tool for developing and evaluating text summarization models that integrates with Docker plugins and provides visual analysis of models’ strengths and weaknesses. |
| Outcome: | The proposed model and evaluation measures can be easily integrated as Docker-based plugins and provide insights into the models’ strengths and weaknesses. |
Extending Multi-Document Summarization Evaluation to the Interactive Setting (2021.naacl-main)
Copied to clipboard
| Challenge: | Existing approaches to interactive summarization are incomparable and divergent . a key gap in the development and adoption of interactive summaries is the lack of evaluation methodologies and benchmarks for meaningful comparison of systems. |
| Approach: | They propose an end-to-end evaluation framework for interactive summarization based on expansion-based interaction . framework includes procedure of collecting real user sessions, evaluation measures relying on summarizing standards, but adapted to reflect interaction. |
| Outcome: | The proposed evaluation framework is based on evaluations of baseline implementations and is available publicly as a benchmark. |
Increasing Faithfulness in Knowledge-Grounded Dialogue with Controllable Features (2021.acl-long)
Copied to clipboard
| Challenge: | Existing systems that strive to be informative teachers are difficult to build . knowledge grounded dialogue systems are difficult because of limited training objectives . |
| Approach: | They propose to train a generative neural dialogue model that is controlled to stay faithful to evidence . they propose to use additional inputs to generate more objective responses . |
| Outcome: | The proposed model produces responses that are perceived by humans to be objective and faithful to evidence. |
Quantifying Misattribution Unfairness in Authorship Attribution (2025.acl-short)
Copied to clipboard
| Challenge: | Authorship misattribution can have profound consequences in real life . authors are considered as potential authors in forensic settings . |
| Approach: | They propose a measure to quantify the unfairness of authorship attribution systems . authors find that authors are more likely to be misattributed than others . |
| Outcome: | The proposed model shows that some authors are more likely to be misattributed than others. |
Automatic Estimation of Simultaneous Interpreter Performance (P18-2)
Copied to clipboard
| Challenge: | Existing methods to predict interpreter confidence and the adequacy of the interpreted message are lacking. |
| Approach: | They propose to extend a QE pipeline to estimate interpreter performance by using five settings in three language pairs. |
| Outcome: | The proposed method can predict interpreter confidence and adequacy over five settings in three language pairs and improves interpretation strategy and evaluation measures. |
Debiasing Isn’t Enough! – on the Effectiveness of Debiasing MLMs and Their Social Biases in Downstream Tasks (2022.coling-1)
Copied to clipboard
| Challenge: | Existing measures for social bias evaluation are inadequate for MLMs to accurately evaluate the social biases in their systems. |
| Approach: | They propose task-agnostic intrinsic and task-specific extrinsic social bias evaluation measures for MLMs that use different methods to re-learn social biases during fine-tuning on downstream tasks. |
| Outcome: | The findings highlight the limitations of existing MLM bias evaluation measures and raise concerns on the deployment of MLMs in downstream applications using those measures. |
The Hitchhiker’s Guide to Testing Statistical Significance in Natural Language Processing (P18-1)
Copied to clipboard
| Challenge: | Statistical significance testing is a standard statistical tool designed to ensure that experimental results are not coincidental. |
| Approach: | They propose a protocol for statistical significance test selection in NLP setups . they propose he proposes a survey of the most relevant tests to help guide the protocol . |
| Outcome: | The proposed protocol includes a survey of the most relevant tests. |
Assisting the Human Fact-Checkers: Detecting All Previously Fact-Checked Claims in a Document (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Recent years have brought us a proliferation of false claims online, which spread fast . fact-checkers have been using automated fact-finding to verify claims . |
| Approach: | They propose a system that can detect claims that can be fact-checked by a given database . they create a manually annotated document dataset and propose evaluation measures . |
| Outcome: | The proposed system achieves sizable performance gains over strong baselines. |
Concept-Best-Matching: Evaluating Compositionality In Emergent Communication (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing evaluation methods do not expose compositionality of emergent communication . compositionality is a trait that enables the construction of complex meanings from the meaning of parts. |
| Approach: | They propose to find best-match between emergent words and natural language concepts to assess compositionality of emergentic communication. |
| Outcome: | The proposed algorithm provides a global score and translation-map between emergent words and natural language concepts. |
Evaluating Evaluation Measures for Ordinal Classification and Ordinal Quantification (2021.acl-long)
Copied to clipboard
| Challenge: | Ordinal Classification (OC) tasks require ordinal classes, not nominal ones, to be evaluated. |
| Approach: | They use data from the SemEval and NTCIR communities to clarify evaluation measures for Ordinal Classification and Ordinal Quantification tasks. |
| Outcome: | The evaluation measures for Ordinal Classification (OC) and Ordinal Quantification (OQ) tasks are ordinal, not nominal. |
Linguistic Appropriateness and Pedagogic Usefulness of Reading Comprehension Questions (2020.lrec-1)
Copied to clipboard
| Challenge: | Existing evaluation measures for automatic generation of reading comprehension questions focus on linguistic quality only, ignoring educational value and appropriateness of questions. |
| Approach: | They propose a new evaluation scheme where questions are structured in a hierarchical way . they also create and evaluate two new evaluation data sets for Basque and German . |
| Outcome: | The proposed evaluation scheme can be applied, but expert annotators are needed. |
An Evaluation Method for Diachronic Word Sense Induction (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods to detect semantic shifts across time are based on time-stamped annotated biomedical data . dynamic behaviour of words contributes to semantic ambiguity, which is a challenge in many NLP tasks. |
| Approach: | They propose an evaluation method based on large-scale time-stamped biomedical data . they propose a model which represents the temporal dimension of the task . |
| Outcome: | The proposed method is applied to two recent DWSI systems . it provides an in-depth analysis of the models . |
Evaluating Pronominal Anaphora in Machine Translation: An Evaluation Measure and a Test Suite (D19-1)
Copied to clipboard
| Challenge: | Currently, machine translation is performed at the level of individual sentences, in isolation from the rest of the document. |
| Approach: | They propose a dataset that can be used as a test suite for pronoun translation . they propose an evaluation measure to differentiate good and bad pronounce translations . |
| Outcome: | The proposed dataset can be used as a test suite for pronoun translation in English . it covers multiple source languages and different pronouner errors drawn from real system translations . |
Answers Unite! Unsupervised Metrics for Reinforced Summarization Models (D19-1)
Copied to clipboard
| Challenge: | Abstractive summarization approaches based on Reinforcement Learning (RL) have been proposed to overcome classical likelihood maximization. |
| Approach: | They propose to use Reinforcement Learning to learn the model parameters through RL techniques to overcome classical likelihood maximization. |
| Outcome: | The proposed measures favor ROUGE with the additional property of not requiring reference summaries. |
Are we there yet? Encoder-decoder neural networks as cognitive models of English past tense inflection (P19-1)
Copied to clipboard
| Challenge: | linguistics and cognitive science have long debated the cognitive mechanisms needed to account for the English past tense. |
| Approach: | They propose to use an encoder-decoder model to account for the english past tense . they also show that ED models demonstrate humanlike performance in a nonce-word task . |
| Outcome: | The proposed model is unstable across simulations and does not fit to human data . other neural models might do better, but there is insufficient evidence to claim them . |
Revisiting the Evaluation of End-to-end Event Extraction (2021.findings-acl)
Copied to clipboard
| Challenge: | Existing EE research uses the role-averaged evaluation metric, but it is misleading to downstream applications. |
| Approach: | They propose two new evaluation metrics that explicitly penalize wrongly identified event arguments. |
| Outcome: | The proposed evaluation metrics improve the initial evaluation by 10% . the proposed training scheme is better than the existing one, the authors show . |
From Chaos to Clarity: Claim Normalization to Empower Fact-Checking (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Social media posts are noisy and pervasive, resulting in difficult to identify precise and prominent claims that require verification. |
| Approach: | They propose a task called Claim Normalization that decomposes complex and noisy social media posts into more straightforward and understandable forms, termed normalized claims. |
| Outcome: | The proposed model outperforms baselines across evaluation measures and errors. |
A Data-Driven Guided Decoding Mechanism for Diagnostic Captioning (2024.findings-acl)
Copied to clipboard
Panagiotis Kaliosis, John Pavlopoulos, Foivos Charalampakos, Georgios Moschovis, Ion Androutsopoulos
| Challenge: | Diagnostic Captioning (DC) systems receive one or more medical images of a patient, such as X-Rays or Magnetic Resonance Images (MRIs). |
| Approach: | They propose a data-driven guided decoding method that incorporates medical information into the beam search of the diagnostic text generation process. |
| Outcome: | The proposed method improves on two medical datasets and can be used in few- and zero-shot learning scenarios. |
HOLMS: Alternative Summary Evaluation with Large Language Models (2020.coling-main)
Copied to clipboard
| Challenge: | Efficient document summarization requires evaluation measures that can rank a set of systems based on an average score and highlight which individual summary is better than another. |
| Approach: | They propose a hybrid evaluation measure for document summarization called HOLMS that combines both language models pre-trained on large corpora and lexical similarity measures. |
| Outcome: | The proposed measure outperforms ROUGE and BLEU on several extractive summarization datasets for both linguistic quality and pyramid scores. |
SUPER: Evaluating Agents on Setting Up and Executing Tasks from Research Repositories (2024.emnlp-main)
Copied to clipboard
Ben Bogin, Kejuan Yang, Shashank Gupta, Kyle Richardson, Erin Bransom, Peter Clark, Ashish Sabharwal, Tushar Khot
| Challenge: | Large Language Models (LLMs) have made significant progress in writing code, but can they be used to reproduce results from research repositories? |
| Approach: | They propose a benchmark to evaluate the capability of Large Language Models to reproduce results from research repositories. |
| Outcome: | The benchmark aims to capture the realistic challenges faced by researchers working with machine learning and natural language processing repositories. |
Interpreting Topic Models in Byte-Pair Encoding Space (2025.coling-main)
Copied to clipboard
| Challenge: | Byte-pair encoding (BPE) is a popular method of tokenizing valid words onto a token space V b with a predetermined fixed size, and handling out-of-vocabulary words, breaking words into smaller tokens. |
| Approach: | They propose to interpret the recovery of valid words from these tokens as a ranking problem and apply existing evaluation measures to topic sets. |
| Outcome: | The proposed model interprets the recovery of valid words from these tokens as a ranking problem and applies existing evaluation measures. |
Keyphrase Generation: Lessons from a Reproducibility Study (2024.lrec-main)
Copied to clipboard
| Challenge: | Reproducibility studies are used to verify the validity of a scientific method, but what else can we learn from such experiments? |
| Approach: | They use Keyphrase Generation to examine reproducibility under different conditions . they draw conclusions on state of the art in KPG and provide guidelines for researchers . |
| Outcome: | The proposed models are compared under the same or varied conditions and provide guidelines for reporting results in a more comprehensive manner. |
A Comparative Study on the Impact of Model Compression Techniques on Fairness in Language Models (2023.acl-long)
Copied to clipboard
| Challenge: | Existing literature demonstrates that compressing deep learning models could affect their fairness. |
| Approach: | They evaluate pruned, distilled, and quantized language models to assess their fairness . they also examine the impact of using multilingual models and evaluation measures . |
| Outcome: | The proposed methods can reduce the fairness of language models by reducing their complexity and reducing the cost of training and deployment. |
SCURank: Ranking Multiple Candidate Summaries with Summary Content Units for Enhanced Summarization (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing ranking strategies for large language models suffer from instability and lack of information content. |
| Approach: | They propose a framework that enhances summarization by leveraging Summary Content Units (SCUs) they investigate the effectiveness of SCURank in distilling summaries from multiple LLMs . |
| Outcome: | The proposed framework outperforms traditional metrics and LLM-based ranking methods in summarization tasks. |